👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

Thanks for the thorough review @alvinttang! Addressing each point:

1. Thread safety — Added a comment in ExpertLRUCache noting the single-threaded assumption (68c81df). You're right that if vLLM ever supports concurrent forwards with shared weights this would need a lock.

2. Synchronous H2D copies — Agreed, this is the main latency bottleneck. Async H2D with double-buffered CUDA streams (the "DBO scheduling" from RFC #33869) is the top item in the planned PR 2. Mentioning it here so it's on record.

3. Persistent mapping tensor — Implemented in 68c81df. _mapping is now a persistent [num_experts] GPU int32 tensor, updated in-place at each miss. Eliminates the per-call CPU allocation + Python loop + H2D transfer from the hot path.

4. Bias bypass — Guard added in 68c81df: _maybe_init_expert_lru_cache() checks moe_config.has_bias and logs a warning + returns early, so the cache is disabled rather than producing wrong results. A follow-up PR can wire bias tensors through (they're small, so CPU-pinning them is trivial) when a bias-using MoE model needs offloading.

5. enforce_eager guard — In the code since 68c81df. From FusedMoE.__init__():

if self._moe_expert_cache_size > 0 and (
    not vllm_config.model_config.enforce_eager
):
    logger.warning(
        "moe_expert_cache_size requires --enforce-eager; ..."
    )
    self._moe_expert_cache_size = 0

The cache is silently disabled (not just warned) when enforce_eager=False.

6. Memory accounting — Valid concern. The GPU profiler won't see CPU-pinned allocations, so it will over-allocate KV cache against memory that expert weights no longer occupy. This is actually a benefit (more KV cache headroom), not a hazard — the expert weights are no longer on GPU. But you're right that if someone relies on gpu_memory_utilization for precise sizing, the accounting is off. I'll add a note to the PR description.

7. Tests — 16 unit tests in tests/kernels/moe/test_expert_lru_cache.py (618392a), covering: hit/miss counters, LRU eviction correctness, slot remapping, GPU buffer content post-eviction, dtype preservation, CPU pinned backing store, FP8 per-slot scale buffering, and the no-scales path. Edge case capacity >= num_experts (no eviction pressure) is implicitly covered by _free_slots never emptying in those scenarios.

Hi @e1n00r, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

Hi @e1n00r, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

Also check this paper: https://arxiv.org/html/2410.17954v1

Instead of LRU, they load with a predictor:

"ExpertFlow consists of three key components: the Routing Path Predictor, the Expert Cache Engine, and the Token Scheduler.

Leveraging the three synergistic components of our system, ExpertFlow achieves an average GPU memory savings of 75.4%, with peak savings reaching up to 93.72%, compared to GPU-only solutions. Furthermore, ExpertFlow attains an expert cache hit ratio of up to 91.96%, improving the hit ratio by an average of 27.65% over the LRU caching strategy. Additionally, ExpertFlow delivers a 2 to 10 times increase in inference speed."

Also check this paper: https://arxiv.org/html/2410.17954v1

Instead of LRU, they load with a predictor:

"ExpertFlow consists of three key components: the Routing Path Predictor, the Expert Cache Engine, and the Token Scheduler.

Leveraging the three synergistic components of our system, ExpertFlow achieves an average GPU memory savings of 75.4%, with peak savings reaching up to 93.72%, compared to GPU-only solutions. Furthermore, ExpertFlow attains an expert cache hit ratio of up to 91.96%, improving the hit ratio by an average of 27.65% over the LRU caching strategy. Additionally, ExpertFlow delivers a 2 to 10 times increase in inference speed."

If I do that we just made powerinfer again, which is a well established solution in its own right.
Also this would necessitate training of predictor models (just as powerinfer does).
If I were the target audience I would just use that backend instead.
The point of vLLM for me at least, is its scalability and wide support, adding this requirement would make the feature nigh useless.
perhaps a middle ground? something that learns on the fly?
Anyway, I would push anything of that magnitude to another PR.

Well, apparently there's quite a few options here:

1. Frequency–recency hybrid (statistical scoring)
   A. Least Cache Priority (LCP) / exponential decay scoring. Combines frequency (μ) and recency gap (ν) into a single score
   B. ARC (Adaptive Replacement Cache)
   C. LFU

2. Reuse-distance / stack-distance models
   D. LIRS (Low Inter-reference Recency Set)
   E. Reuse-distance–based admission control (general approach)

3. Structure-aware (MoE-specific statistical policies)
   F. Layered-LRU (LLRU)
   G. Miss-rate–constrained caching (global statistical control)

4. Partial / fractional caching (statistical resource allocation)
   H. Bit-sliced / fractional expert caching (DBSC)

5. Admission-control + statistical filtering
   I. Probabilistic admission (TinyLFU-style ideas applied to MoE)

6. Hybrid statistical + constraint-based approaches
   J. Multi-tier statistical caching

But these are not better than Predictor-based systems (e.g., ProMoE, ExpertFlow) and Learned replacement (e.g., FlashMoE ML policy).

Strong non-ML alternatives:
Best general: ARC, LIRS
Best MoE-specific: LLRU, LCP
Most novel: bit-sliced / fractional caching
Most promising direction (non-ML): Score-based caching

Of course, that's all for another PR, it's important to at least get this caching strategy ball rolling - the possible speedups seem to be massive. Maybe it would be nice to make the strategy pluggable?

Gemma 4 26B-A4B-it validation (128 experts × top-8, 30 layers)

Tested the rebased fork on google/gemma-4-26B-A4B-it — a very different MoE architecture from Nemotron:

Results (cache_size=8, RTX PRO 6000 Blackwell)

Expert LRU cache enabled for language_model.model.layers.0.moe.experts: 8/128 experts cached on GPU.
...
Expert LRU cache enabled for language_model.model.layers.29.moe.experts: 8/128 experts cached on GPU.
Model loading took 8.62 GiB memory and 211.945657 seconds

Required fixes for Colab/Jupyter

Had to add a try/except in vllm/utils/system_utils.py:suppress_stdout() because sys.stdout.fileno() raises io.UnsupportedOperation in ipykernel subprocesses (EngineCore runs as a separate process that inherits Jupyter's stdout). Fix: commit dab55b3.

Also quantized with PolarQuant Q5

Quantized all 7,680 MoE expert weights (3D nn.Parameter tensors) + 427 nn.Linear layers with PolarQuant Q5, saving codes to HuggingFace. Download reduced from 51.6 GB → 26.9 GB.

Note on LFRU

Looking forward to testing the LFRU policy on this model. With 128 experts × 30 layers and top-8 routing, the deep-layer cache starvation you described for GPT-OSS-20B should be even more pronounced here. I've implemented an initial LFRU version in the fork (commit 7a19e4d) — happy to coordinate on this.

@caiovicentino — the LFRU results confirm what we measured independently. We have LFRU and 6 other policies implemented and benchmarked in tinyserve.

Based on your validation (LFRU cache=8 exceeds LRU cache=16 in hit rate) and our own benchmarks (deep-layer starvation eliminated, +5-50% throughput), we've updated the PR to ship LFRU as the built-in eviction policy instead of LRU. No config field, no pluggable framework — just the better algorithm baked in. Keeps the PR focused.

The Gemma 4 validation (14.8 tok/s, 8.62 GB) is strong — two models on two architectures with correct output.

For the rebase: we've rebased onto current main using your conflict resolutions (co-authored). One clean commit, 14 files, no workflow changes.

For experimentation: if you want to try ideas freely — buddy substitution, CPU-on-miss, dynamic VRAM rebalancing, imatrix cache seeding, per-layer cache budgets — tinyserve is the playground. It's ~7K LOC Python with 340 tests, all of these features implemented and benchmarked. Alternatively, we could set up a shared experimental branch on the vLLM fork for testing concepts before they go into a PR. Either way works — happy to coordinate.

@mgoin — could you add the ready label to trigger CI? All pre-commit checks are now passing (mypy clean, ruff formatted). Two independent validations in the comments: Nemotron-Cascade-2-30B-A3B (7.6 GB VRAM, 15+ tok/s) and Gemma-4-26B-A4B-it, both with correct output.

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @e1n00r.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@e1n00r — We rebased the expert offloading code on top of the latest upstream/main (70406eb, April 7th) and confirmed it works end-to-end with a CompressedTensors INT4 MoE model.

Test results (Qwopus-MoE-35B-A3B INT4 CT, RTX PRO 6000 Blackwell 102 GB):

The LFRU cache is 1.58x faster than all-in-GPU — better memory locality with 8 hot experts vs 256 scattered. Coherent output confirmed (not garbage).

Our rebased fork: https://github.com/caiovicentino/vllm-expert-offload (branch main, commit f324bd9)

The rebase resolved one conflict in unquantized_fused_moe_method.py (new XPU branch added upstream). All 9 commits apply cleanly. Happy to help with the rebase on your side — you can cherry-pick from our branch if useful.

Co-authored-by: Caio Vicentino caiovicentino@users.noreply.github.com
Co-authored-by: Claude noreply@anthropic.com

Benchmark charts for the INT4 MoE test above:

Model card with full details: https://huggingface.co/caiovicentino1/Qwopus-MoE-35B-A3B-PolarQuant-Q5

PPL 6.56 on full WikiText-2 (295K tokens) — virtually identical to BF16 baseline. The LFRU expert cache with only 8 hot experts is faster than loading all 256 into GPU.

Hi @e1n00r @caiovicentino ! Thank you for your excellent work and test results. I have a small problem though: when the cache is overflowed during prefill, the current behaviour is to keep the suffix of the required expert list, which consists of ones with the greatest ids since the list sorted by unique(). I doubt that whether the experts kept this way are "most likely to be needed in upcoming decode steps", and I am wondering how it will impact the model performace without the results of the discarded experts.

@tkj666 great catch — this is a real limitation and worth being explicit about.

You're correct: torch.unique() returns the tensor sorted, so slicing the suffix keeps the highest expert IDs, which has no relationship to access order. It's arbitrary eviction during prefill.

Why it ended up this way

During prefill there's no LRU history yet — no prior tokens have touched the experts, so we have zero signal about which to keep. We defaulted to "keep the suffix of unique()" as a placeholder without claiming it's optimal. The LRU signal only starts accumulating once decode begins.

The real issue

For short prefills (len(unique_experts) ≤ cache_size) this never matters — everything fits. For long prefills where unique experts exceed cache size:

1. Prefill thrashes, forced to reload overflow experts repeatedly
2. The experts that survive are arbitrary high-ID ones
3. Decode starts with a cache populated by essentially random experts
4. LRU gradually corrects to real decode patterns

Better policy: prefill-access-order LRU

The right fix is tracking expert access order during prefill in a small ring buffer and using that as the initial LRU state when decode begins. Turns "arbitrary prefill state" into "actual recency from prefill traffic" at near-zero cost.

Benchmark plan

We want to quantify the quality impact before picking a policy. We're planning to compare on a long-prompt workload with moe_expert_cache_size=4:

Thanks for flagging this — it matters for long-context / small-cache workloads exactly like you describe. Will follow up on this thread when we have numbers.

@tkj666 @caiovicentino — thanks both. The fix is now in 1a8df90.

Digging into it we found the suffix-of-unique() behavior is actually a correctness bug, not just a quality-of-eviction issue:

_mapping is initialized as torch.zeros(num_experts, dtype=int32) and is only updated for experts that are loaded into a slot. On overflow, the dropped experts keep whatever value _mapping had — slot 0 on a cold cache, or a previously-evicted expert's old slot on a warm cache. The kernel then reads _mapping[dropped_id], indexes into the GPU buffer, and computes with a different expert's weights for every token routed to a dropped expert. The tokens still get a real matmul — just with the wrong weights, and the wrong activations propagate downstream.

Given that, any truncation policy (suffix / prefix / random / prefill-access-order) has the same defect unless we also fence off the dropped experts in _mapping. For PR1 we went with fail-fast:

if len(unique_ids) > self.capacity:
    raise RuntimeError(
        f"CachedWeightProvider: {len(unique_ids)} unique experts requested "
        f"but --moe-expert-cache-size={self.capacity}. "
        f"Set --moe-expert-cache-size >= {len(unique_ids)}."
    )

The prefill-LRU seeding @caiovicentino sketched is the right direction for a follow-up PR alongside chunked-prefill-aware loading — once the kernel has a way to skip tokens whose expert is not resident, the policy question becomes meaningful. Until then, "fit all unique experts in capacity, or error" is the only correct option.

Practical guidance for hitting this: either set --moe-expert-cache-size >= num_experts (all experts GPU-resident — cache disabled at steady state, still saves the CPU→GPU load latency on cold starts), or reduce --max-num-prefill-tokens so a single chunk stays within capacity unique experts.

@e1n00r — I think that we can make prepare() to return a generator of a set of at most capacity experts, and call the kernel over every set before combining the results during prefilling. I suppose that the kernel supports masking out some of the experts like in EP cases, though I haven't really looked into it. The described procedure described above is virtually EP with each replica executing one by one on the same gpu.

Empirical validation

Tested "partition experts into N groups, call fused_experts once per group with expert_map[id]=-1 for out-of-group, accumulate partials" on an RTX PRO 2000 Blackwell. Results across shapes from (M=4, E=8, top-2) through (M=1024, E=32, top-4) and 2/4/8/16 groups: bitwise equal to single-call baseline at smaller shapes; at larger shapes the difference is pure dtype rounding from non-associative bf16 summation — measured ~6.5e-3 rel err for bf16, ~8e-4 for fp16, ~1e-7 for fp32, each exactly matching the mantissa floor. The slot-space decomposition that CachedWeightProvider would actually use (sliced w1[:cap]/w2[:cap] per group with remapped slot ids) is also bitwise equal to the baseline.

The constraint: use fused_experts_impl, not the modular kernel

The provider currently routes through TritonExperts.apply (modular kernel). That path is incompatible with cross-call output accumulation because of modular_kernel.py:1080-1091: output and workspace13 are both views into common_workspace, and WorkspaceManager in v1/worker/workspace.py keeps one persistent buffer per ubatch. Call N+1's activation writes intermediate_cache2 (which aliases output storage via workspace13), clobbering call N's output. A loop with output += partial_g on the modular path silently corrupts the accumulator.

The non-modular fused_experts_impl path is the correct target: it allocates cache13/cache2/out_hidden_states fresh per call with no shared workspace, already calls intermediate_cache3.zero_() when expert_map is not None (fused_moe.py:1862-1863), and uses ignore_invalid_experts=True in moe_align_block_size to drop -1-routed tokens at sort time rather than writing zeros through the kernel. Empirically verified: each fused_experts(..., inplace=False) call returns an independently-allocated tensor (distinct data_ptr()), so outer accumulation across calls is safe.

Sketch for forward_native:

if provider is not None:
    chunks = provider.prepare_chunked(topk_ids)  # yields one chunk when unique fits
    if len(chunks) == 1:
        r = chunks[0]
        return fused_experts(x, r.w1, r.w2, topk_weights, r.topk_ids,
                              inplace=False, expert_map=r.slot_map)
    acc = torch.zeros_like(x)
    for r in chunks:
        acc += fused_experts(x, r.w1, r.w2, topk_weights, r.topk_ids,
                              inplace=False, expert_map=r.slot_map)
    return acc

Cost model: PCIe transfer dominates

Measured on the target hardware at M=512, K=2048, N=2048, E=32, top-4, 24 MB per bf16 expert (w13+w2):

Kernel launch overhead is secondary: +6% for 2 groups, +15% for 4, +35% for 8 at M=1024 prefill (5-6 Triton launches per call × ~100 μs each). The 15x slowdown in the cold-swap case is entirely PCIe bandwidth — total bytes transferred are identical regardless of how you partition the groups.

Practical consequence for the design: mini-EP is the correct fallback when VRAM cannot fit all experts, and --moe-expert-cache-size >= num_experts remains the recommended path whenever it fits. The overhead grows linearly with the number of new experts per layer, so the design should prefer a high cache hit rate over a clever grouping strategy.

Scope: PR2 alongside async H2D

Holding this out of PR1 for three reasons:

1. PR2 already restructures prepare() and the forward loop for the async copy stream. Mini-EP and async prefetch share the same API surface (prepare_chunked() with optional pre-fetched slots), so designing them together avoids an architectural carve-out that a PR1 bolt-on would require.

2. Routing mini-EP through fused_experts_impl means the provider path diverges from the modular kernel infrastructure for shared_experts / quant_config / prepare_finalize dispatch. That's a deliberate architectural choice deserving its own design discussion with @mgoin, rather than a late addition to a month-old PR.

3. Two open questions require real-model runs:

   • Does try_get_optimal_moe_config cache-hit across groups when effective num_valid_tokens changes per group? A miss means autotune re-runs every group, destroying first-forward latency.
   • Does bf16 accumulation drift over 30-60 layers × hundreds of forward passes stay within atol=2e-2? Microbenchmark drift is at the mantissa floor; real-model drift under repeated reductions is untested.

Happy to prototype this on a follow-up branch after PR1 merges — your framing ("mini-EP with each replica executing one by one on the same GPU") is the right mental model, and the two non-obvious implementation points are (a) the non-modular routing required to avoid the workspace aliasing, and (b) hoisting unique() out of the per-group loop to pay the D2H sync once per forward rather than N times.

Thanks for the rigor on this — the workspace aliasing finding alone is the kind of latent bug that would survive unit tests and only surface in real-model accumulation. Calling it a correctness defect rather than a "be careful" footnote is the right framing.

A few notes on each section:

Empirical validation. The bitwise-equal-at-small / mantissa-floor-at-large pattern is exactly the equivalence story you can defend in a PR description without hedging.

fused_experts_impl vs modular kernel. I hadn't connected the intermediate_cache2 / workspace13 aliasing to mini-EP. Routing through fused_experts_impl is the right call, and ignore_invalid_experts=True already in moe_align_block_size makes the slot-map decomposition much cleaner than I'd sketched.

PCIe dominance. Matches what we saw on Nemotron-Cascade-2-30B-A3B and Gemma-4-26B-A4B. The LFRU validation reproduced the same early-layer-hot / deep-layer-starved pattern you described on GPT-OSS-20B, and the hit-rate gains were the dominant lever — bigger than any grouping cleverness we tried. Your 15x cold-swap measurement reinforces that "fit, or fail fast" is the only honest PR1 contract.

PR2 scope. Agree on holding it out. On the two open questions:

1. Autotune cache invalidation across groups — I can run this on Nemotron-Cascade-2-30B-A3B and Gemma-4-26B-A4B-it (the two architectures already in this PR's validation set). They stress autotune differently — Nemotron's deep layer count surfaces shape-driven misses, Gemma's smaller per-layer expert count surfaces expert-id-driven ones.

2. bf16 accumulation drift over 30-60 layers — same models, full forward sweep with running max-rel-err against a fp32 reference. If drift stays below 2e-2 after 100 forward passes, the bound holds. Above that, chunked-prefill becomes load-bearing.

Happy to do both on whatever branch you set up after PR1 merges. Ping me when there's a target commit.